Value Iteration in a Class of Average Controlled Markov Chains with Unbounded Costs: Necessary and Sufficient Conditions for Pointwise Convergence
نویسندگان
چکیده
This work concerns controlled Markov chains with denumerable state space, (possibly) unbounded cost function, and an expected average cost criterion. Under a Lyapunov function condition, together with mild continuity-compactness assumptions, a simple necessary and sufficient criterion is given so that the relative value functions and differential costs produced by the value iteration scheme converge pointwise to the solution of the optimality equation; this criterion is applied to obtain convergence results when the cost function is bounded below or bounded above. CONTROLLED MARKOV CHAINS; AVERAGE COST CRITERION; LYAPUNOV FUNCTION CONDITION; VALUE ITERATION SCHEME; POINTWISE CONVERGENCE; NECESSARY AND SUFFICIENT CONDITIONS AMS 1991 SUBJECT CLASSIFICATION: PRIMARY 93E20; 90C40
منابع مشابه
Drift and monotonicity conditions for continuous-time controlled Markov chains with an average criterion
In this paper, we give conditions for the existence of average optimal policies for continuous-time controlled Markov chains with a denumerable state–space and Borel action sets. The transition rates are allowed to be unbounded, and the reward/cost rates may have neither upper nor lower bounds. In the spirit of the “drift and monotonicity” conditions for continuous-time Markov processes, we pro...
متن کاملUniform Convergence of Value Iteration Policies for Discounted Markov Decision Processes
This paper deals with infinite horizon Markov Decision Processes (MDPs) on Borel spaces. The objective function considered, induced by a nonnegative and (possibly) unbounded cost, is the expected total discounted cost. For each of theMDPs analized, the existence of a unique optimal policy is assumed. Conditions that guarantee both pointwise and uniform convergence on compact sets of the minimiz...
متن کاملNecessary and Sufficient Conditions for Global Geometric Convergence of Block Gauss-seidel Iteration Algorithm Applied to Markov Chains
Convergence properties of the block Gauss-Seidel algorithm applied to ergodic Markov chains are discussed in this paper. This algorithm is one of the most prevalent methods for computing ergodic probability vectors of large-scale Markov chains. We will provide necessary and sufficient conditions for global geometric convergence of this algorithm. To apply this algorithm, the state space of a Ma...
متن کاملOn the optimality equation for average cost Markov decision processes and its validity for inventory control
As is well known, average-cost optimality inequalities imply the existence of stationary optimal policies for Markov decision processes with average costs per unit time, and these inequalities hold under broad natural conditions. This paper provides sufficient conditions for the validity of the average-cost optimality equation for an infinite state problem with weakly continuous transition prob...
متن کاملTime and Ratio Expected Average Cost Optimality for Semi-Markov Control Processes on Borel Spaces
We deal with semi-Markov control models with Borel state and control spaces, and unbounded cost functions under the ratio and the time expected average cost criteria. Under suitable growth conditions on the costs and the mean holding times together with stability conditions on the embedded Markov chains, we show the following facts: (i) the ratio and the time average costs coincide in the class...
متن کامل